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(57) Abstract 

A computer based interactive system branches among a variety of inputs from a variety of different sources (30. 38. 42) during th 
same interacUve session. This interactive system integrates switching (10) of full-motion video, graphics and audio from various source 
simultaneously during the show. 
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SYSTEM FOR PROVIDING AN INTERACTIVE PRESENTATION 
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RELATT.n APPIlrAT |n|^ff; 

This is . conanua«o„-in.parl of app.icaUon Serial No. 08/443,607 filed 
May 18, 1995, which is cc„.„„a.ion-i„.par. of application serial no. 08/166 608 
filed December 13, 1993 which in fu. is a conhnuafon of application serial no 
07/797,298, filed November 25. 1991, now abandoned. TT,e interactive 
television program is of a type disclosed in commonly copendmg U.S. Paten, 
Applicae.on Serial No. 08/289,499, filed August 12, 1994, and in U.S. Paten, 
Nos. 4347,700, 4,507,680, 4,573,072, 3,947,792, 4.602,279, 4,264,925, and 4,264,924. 

BACKGRDl iMn r^T thf IMVp Mj,^|>| 
Interacive v.deo and audto presentation systems are curren.lv being 
introduced into the entertammen, and educat.onal industr.es. A prommen. 
-teractive technology that has been app„ed successfully in these indastr.es is ' 
based on prov.dmg interactivity in a one-way system through the provision of 
multiple parallel channels of informafon. For example, commonlv owned 
Freeman e. al. patents, U.S. paten, nos. 4,264,925 and 4,264,924, which prov.de 
both audio and video mteractivity, disclose in.eract.ve television systems 
«here switching among muhiple broadcast or cable channels based on viewer 
selections provides an interactive capability. 

These systems have been enhanced to include memory functions using 
computer logic and memory, where selection of system responses played to the 
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viewer are based on the processing and storage of subscriber responses, as 
disclosed in Freeman patent, U.S. patent no. 4,507,680. 

The benefits of providing interactivity through the use of different 
audio responses is disclosed in Freeman, U.S. patent nos. 4,847,698, 4,847,699 
and 4,847,700. These television systems provide a common video signal 
accompanied by several synchronized audio channels to provide content 
related user selectable responses. The audio signals produce different audio 
responses, and in some cases, these are syllable synched to a first audio script 
and to the video signal (such as to a person or character on a display), 
providing the perception that the person's or character's mouth movements 
match the spoken words. 

Interactivity is brought to the classroom in the Freeman U.S. patent 
application serial no. 08/228,355. The distance learning system claimed in this 
application enhances the classroom educational experience through an 
innovative use of interactive technology over transmission independent 
media: When an instructor, either broadcast live on video or displayed from 
videotape, asks a question, (each and every student responds, preferably by 
entering a response on a remote handset, and each student immediately 
receives a distinct and substantive audio response to his or her unique 
selection. The individualization of audio response from the interactive 
program is a major aspect of the invention. 

Individualization of audio is brought to the home based on the 
technology disclosed in Freeman U.S. patent application serial no. 

( 
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08/228,355. This system provides a program that can be watched on any 
conventional television set or multimedia computer as a normal program. 
But if the viewer has a special interactive program box connected to the 
television, he or she can experience a fully functional interactive program. 
Each interactive viewer enjoys personalized audio responses and video 
graphics overlayed on the screen. The interactive program can be provided to 
television sets or to computers by cable, direct broadcast satellite, television C 
broadcast or other transmission means, and can be analog or digital. Unlike 
previous interactive systems, this application covers a system that subtly 
introduces the interactive responses to the viewer throughout the program. 
This enhanced interactivity is provided through the use of "trigger points" 
spread throughout the program. Trigger points occur at designated times and 
result in the program content being altered to present individual attention to 
the particular viewer. 

However, what is needed is an interactive personalization provided via C 
an interactive multimedia computer. Furthermore, a system is needed that 
provides not only the ability to branch amongst parallel transmitted 
datastreams, but also, the capability to seamlessly integrate input from other 
media, such as CD-ROMs and laser disks, into the presentation. What is 
needed is a computer-based system for branching between a variety of inputs 
during the same interactive session including full-motion video, computer 
graphics, digital video overlays and audio. 
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SUMMARY OF THE INVENTION 

. The ACT\^ system is based upon branches which occur in the course of 
the full-motion video. Branches may be to other full-motion video segments, 
• to graphics which are integrated into the video, and /or to audio segments 
which are integrated into the show. 

Sometimes, the ACTV system will act upon the user's response 
immediately; other times, it will utilize ACTV's unique "trigger point" concept 
to act upon the response later. ACTV's technology enables the computer to 
"remember" the user's responses and integrate them into the video and audio 
at a later point. Regardless of whether the action is taken as a result of the 
user's response immediately or later, it is done seamlessly. 

ACTV's television technology provides the capability to seamlessly 
branch among multiple video and audio sources. ACTV's computer 
technology provides the ability to seamlessly branch not only among the 
multiple video and audio channels, but also to seamlessly integrate input from 
other media, such as CD-ROM's, laser disks, hard disks, and remote servers, 
connected via the Internet or another network, into the show. 

During a television-based ACTV interactive session, the system will 
branch among either multiple television channels or multiple audio sources, 
depending upon the type of implementation. By contrast, during a computer- 
based interactive session, branches may be among a variet)' of inputs from a 
variety of. different sources during the same interactive session: full-motion 
video, computer graphics and audio. Since the computer provides the 
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capability to process data from various multimedia inputs simultaneously, 
ACTV technology can integrate seamless switching of hill-motion video, 
graphics and audio from various sources simultaneously during the show. 
The computer-based ACTV implementation is therefore much more flexible 
than the television-based ACTV implementation. 

It also provides the user with the capability to interact with the show 
utilizing a variety of input devices. Not only can the user interact with the ^ 
show by pressmg a multiple-choice button, but the interaction can also take the 
form of entry via the range of multi-sensory devices available on the 
computer, including mouse entry, full-motion pen entry and touch screens. 
This integration of various input and storage devices is particularly valuable in 
an educational environment, since it provides students with the ability to 
participate in their lessons in a variety of ways. The computer can both store 
the interactions for future reference and also transmit them to the teacher, via 
either a computer network or, in a distance learning setting, via a telephone C 
network. 

An ACTV interactive session can integrate full-motion video with user 
input at the same time. For example, the full-motion video may be playing on 
the screen, while the user is drawing a diagram in a comer of the screen. Thus, 
the video and audio may provide real-time input which the user is applying 
during the session on the same computer monitor. 
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DESCRIPTION OF THE DRAWINGS 

. Figure 1 is a diagram of an interactive computer workstation, receiving 
inputs from television broadcasts and /or local storage devices. 

Figure 2 is a diagram of an interactive computer workstation which 
5 receives its input primarily from television broadcasts. 

Figure 3 is a diagram of an interactive computer workstation which 
(" receives its interactive programs entirely from local storage, rather than 

television broadcasts. 

Figure 4 is a diagram of an interactive network for interactive 
10 processing. 

Figure 5 is a diagram of an interactive computer system, receiving 
inputs from a multichannel cable transmission and showing outputs via a 
conventional television monitor. 

Figure 6 is a block diagram of one interactive computer workstation 
^ 15 embodiment to achieve seamless switching between video signals. 

Figure 7 is a block diagram showing an alternative interactive computer 
work station embodiment to achieve seamless switching between video 
signals. 

Figure 8 is a block diagram showing another alternative to achieve 
20 seamless switching between video signals. 

Figure 9 is a time diagram showing a representation of trigger points and 
corresponding alternative audio, video or graphics segments, one of which is 
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selected for presentation to the subscriber immediately after the execution of a 
trigger point function. 

Figure 10 is a diagram of an interactive computer work station 
embodiment for branching amongst multiple audio segments in a single video 
5 channel embodiment, where the interactive audio and data elements are 
embedded in the video channel. 

Figure 11 is a diagram of a second mteractive computer work station ^ 
embodiment for branding amongst multiple audio segments in a single video 
channel embodiment, where the interactive audio segments are sen, in the 
10 SAP audio channel. 

Figure 12 is a diagram of a third interactive computer work station 
embodiment for branching amongst multiple audio segments in a single video 
channel embodiment, where two tuners are employed; the first tuner for ' 
tuning to and demodulating the standard video and audio signal and the 
second of wMch is for demodulating a secondary analog carrier comprising ( 
modulated serial digital audio segments. 

Figure 13 is a diagram of a fourth interactive computer work station 
embodiment for branching amongst multiple audio segments in a single video 
channel embodiment, also comprising two tuners, but with a digital 
demultiplexer configuration for demultiplexing the digital audio stream into n 
paraUel digital audio channels, wherein the n parallel digital audio channels 
are time division multiplexed at the head-end and transmitted as a separate 
digital audio stream. 
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PREFERRED EMBODIMENT 
As shown in Figure 1, the present invention is a computer based system 
for receiving a fully interactive program, allowing subscribers to interact with 
5 the program through the use of a keypad and personal computer. 

Alternatively, the multiple video/audio datastreams may be received from a 
broadcast transmission source or may be resident in local or external storage 
including CD ROM, video datatape, etc., as discussed below. 

The interactive computer, 6 uses an interactive program delivery system 
10 with any transmission means including satellite, cable, wire or television 
broadcast to deliver the interactive program (hereinafter "composite 
interactive program") from a centralized location, or operations center, for 
distribution to subscribers in their homes. The program may be broadcast live 
from the operations center. For example, live sporting events with added 
15 interactive elements can be broadcast from the operations center. Such live 
interactive elements could be different camera angles, slow motion video, etc. 
Alternatively, the program can be produced off-line and stored in a program 
storage means at the operations center. Furthermore, the program can be 
produced and stored locally at the remote site on CD ROM or some other 
20 transferrable storage device such as digital or audio videotape, or laser disk. 

• An interactive presentation can comprise branching amongst full 
motion video, computer graphics and audio, with the interactive elements 
either received over a transmission media or stored locally, or both, all within 
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the same show. As shown in Figure 1, the workstation can branch among 
video segments from television broadcasts, local video servers 38, 42 (such as 
CD-ROMs, laser disks and tape players), still images and audio segments from 
the preceding media, as well as those stored digitally on hard disks 34, and 
segments obtained over networks such as the Internet. 

The present invention, as shown in Figure 1, is a system for processing 
on a computer a fully interactive program allowing users to interact with the ' 
program through a computer input device 22 connected to a standard 
computer system 6, comprising a CPU 108, hard disk 34, audio card 30 and 
monitor 18. The interactive multimedia computer 6 resides in the home of 
the subscriber or elsewhere, such as at a cable headend, as described below. If at 
the home, the interactive computer, 6 is usually located near the subscribers- 
television, if connected to the television set. Preferably, any of the multimedia 
computer embodiments, discussed below, comprise a video demodulator 
board, a keypad for entering subscriber selections, a data extractor board 46 (for ( 
extracting data from the vertical blanking interval from the video signal(s), 
temporary and permanent data storage, a modem 14, and a processor 108. 

Broadcast television is received by the video selector 10, which selects 
among various television channels to capture a video signal to be displayed on 
the computer monitor 18. Multiple television channels may be received. 
Figure 2 shows an interactive computer workstation configuration which 
receives its input primarily from television broadcasts. With the technology of 
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the present invention, seamless branching is provided among these television 
channels. 

In the preferred embodiment, interactive programming is also stored on 
Video Source devices 38, 42, as shown in Figures 1 and 3. The Video Sources 
5 38, 42 may be any local storage device which is accessible by the computer, 

including CD-ROMs, laser disks, VCR's and tape players. While Figures 1 and 
3 show only two video sources, there may be any number of such devices. 

When CD-ROM 54 is employed in the present invention, it is a 
component of a unique interactive experience. The present invention utilizes 

10 CD-ROM 54 as one of the multiple input devices. Since branching is always 

seamless in the preferred embodiment, the computer 6 may receive input from 
at least two devices, regardless of whether these sources are random access. 
This is necessary to avoid delays during search periods. While one device is 
playing the video, the other searches for a new branch. When the second 

15 device finds the segment for output display, the other input device searches for 
a new branch. When the second device finds the segment to be shown, the 
branch occurs seamlessly. The apparatus and method for seamless branching 
among various video signals is described in the paragraphs below. 

Segments of the interactive program may also be stored on the 

20 computer's hard disk 34. The segments stored on the hard disk, 34 are usually 
computer graphics, still images or audio segments, which are integrated into 
the presentahon. The format for storage on the hard disk 34 is digital. Any 
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storage device may, however, store any comb.nation of ,uli-n,o.ion video, s,i« 
images, graphics and audio segments. 

As shown in Figures 1-3, the interactive commands are extracted from 
the program by the Command Extractor 46, Ahematively, these commands 
may be stored on an auxiliary storage device such as the hard disk 34. 

The commands are processed by the computer's Central Processing Unit 
(CPU) 108, shown in Figures 1-3. The computer may be an IBM Personal 
Computer (PC, - Compatible, an Apple computer or any other type of standard 
computer workstation. 

The CPU 108 determines what video to display and audio to play based 
upon the inten,ctive commands which it receives. Based upon the commands, 
it plays the appropriate input from its mpu. devices, which are the Video 
Selector 10, V.deo Sources 38, 42 and Hard Disk 34. Aud.o is received and 
processed by the Audio Card 30 which sends audio to Speakers 26 and/or 
15 headphones 50 as shown in Figures 1-3. ^ 
The user interacts with the program through the Input Device 22. This 
device may be a customized keypad, a standard computer keyboard, a mouse to 
"point and click" at selections and also to draw pictures, a touch screen 
(enablmg the user to make a selection by pointing at the screen), a pen-based 
input device (for selecting options or draw pictures), a voice recognition device 
or any other computer input device well known in the art. Furthermore, 
multiple input devices may be accommodated by the system. 



20 
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Regardless of the type of input device 22, user inputs can be utilized by 
the present invention immediately, or at a later time, to result in personalized 
graphics, video and /or audio presentation. For example, the present 
invention utilizes "trigger points," as described below, to enable subsequent 
5 branches among multimedia segments during the show. Additionally, more 
substantive user input, such as pictures and text, may be integrated into the 
(" interactive presentation. These types of user input are particularly useful in 

computer-aided learning applications, since they enable students to participate 
in lessons utilizing various media. The interactive computer 6 provides the 
10 framework to easily integrate the student's multimedia input into the session 
and to transmit the multimedia input to other students and teachers, via 
computer network and /or television broadcast. 

As shown in Figure 4, the interactive system of the present invention 
may operate on a computer network. In this configuration, the program is 
Q 15 processed by the Video Server 70. The programs are sent over the network to 

the Client Starions 58, 62, 66. Any number of client stations may be supported. 
The configuration of each client station is preferably the interactive 
workstation as shown in Figure 3. 

The control for integrating the various multimedia elements is 
20 provided by the ACTV authoring language, a unique set of interactive 

commands to facilitate the interactive process. These commands may either be 
embedded into data portions of full-motion video segments or may reside 
separately on a storage medium such as a Winchester disk. When the 
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commands are embedded within the full-mot.on v.deo (for example, withm 
the vertical blanking interval), the interactions occur as soon as the computer . 
completes the recognition of a command group. When the commands are 
stored separately from the video segments in a digital segment, the timing of 
their execution is based upon "trigger pomts." These trigger points are time 
pomts at which the interactions are to occur, as explained in more detail below. 

The user can view the interactive program either directly using the ^ 
television set 90 or via the computer 94 screen as shown in Figure 5. Figure 5 
is a diagram of an interactive subscriber station, receiving inputs from a 
multichannel cable transmission and showing outputs via either the computer 
94 screen or a conventional television 90 monitor. Cable channels can be 
shown in a window on the PC screen usmg conventional demodulator cards 
such as a WinW card. In this embodiment, a cable set top box receives the 
plurality of analog or digital video/audio signals from the multichannel cable. 
The mteractive multimedia computer 94 also receives the video/audio signals (' 
from the multichannel cable and extracts the data codes, preferably embedded 
in the vertical blanking interval of the video signal(s). The interactive 
computer 94 detector detects and extracts data codes embedded m the data 
stream. These codes are preferably sent to RAM memory and interpreted by 
the main processor. Personalized audio and/or video selection occurs by the 
main processor sending a branching command to the cable set top box. The 
cable set top box processor interprets the command and seamlessly branches to 
the selected video. 
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In the embodiment of Figure 5, the subscriber can receive typical 
conventional video analog programming from a cable headend. Cable systems 
also may be used to convey digital data via a system such as the High-Speed 
Cable Data Service (HSCDS). In a digital system, the subscriber stations may 
receive programming from content servers or Internet Protocol (IP) routers. 
Content servers are typically a combination computer and data storage system 
that stores various types of content from information source providers. These 
providers might provide anything ranging from video games, distance 
learning applications, interactive applications, home shopping applications, 
online magazines and new^spapers, databases, and typical netv^ork and cable 
programming. The IP router, on the other hand, formats, switches, and 
controls the flow of digital data traffic between the cable network and either the 
public switched telephone network (PSTN), the Internet, or commercial on- 
line information services, such as CompuServe and America Online, A 
headend modem modulates the digital data generated by the IP router onto an 
analog carrier signal suitable for transmission downstream to the subscriber. A 
t}q3ical downstream modulation scheme is 64 Quadrature Amplitude 
Modulation (QAM). 

Each downstream transmission reaches the subscriber's house, shown in 
Figure 5, preferably through a tap and drop cable. The cable modem 92 
demodulates the analog carrier and converts the data to a digital format 
readable by the user's PC 94. Alternatively, the cable modem can be replaced by 
an RF demodulator board in the PC, 94. 
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The programming content associated with the present invention may 
reside on either a headend-based or a remote content server or one of the 
storage devices, discussed above (either temporarily or permanently 
downloaded from the content server). Subscribers gain access to the interactive 
programming on the server via an online menu. 

In this digital embodiment, one channel of digitally-compressed video 
content would require about 1.5 Mbps to deliver VCR-quality images to the PC ( 
94, while four channels would require about 6 Mbps. Thus, the interactive 
system of the present invention fits within one 6 MHz channel. At the 
subscriber station, the interactive seamless system could be implemented 
one of the interacti\'e multimedia computers, described below. 



m 



Seamles.s .Switrhing between BroaHr..f K/f.-i^i plp v;h... 
Preferably, the digital video signals are compressed (preferably via MPEG 
15 2 or any other compression scheme) and multiplexed onto a standard NTSC 
signal. The circuitn' in Figures 6-8 below could be implemented on a board 
and inserted into a standard personal computer (PC). A separate 
microprocessor on the interactive board is not necessary for this configuration 
since the standard multimedia PC processor performs the functions of the 
processor 108 shown in Figures 6-8. 

Figures 6-8 show preferred embodiments of the interactive multimedia 
computer 6 of the present invention to enable seamless flicker-free transpai 
switching between the digital video signals on the same channel or different 
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channels. ''Seamless" means that the switch from one video signal to another 
is user imperceptible. These embodiments may be cormected to any 
transmission media or simply cormected to the output of any stand-alone 
storage means (such as CD ROM) for the digitized multiplexed interactive 
program. Preferably, the interactive computer connects to a television or other 
display monitor. To provide this capability, only a digital demultiplexer, 
decompressor(s), frame buffer(s), and sync components are added to the 
conventional multimedia personal computer. These items, and any other 
components, may be connected to the PC processor and storage elements in the 
maimer disclosed in Figures 6-8. 

Figure 6 shows an embodiment which allows for a seamless video 
switch betu'een two or more separate digital video signals. As shown in Figure 
6, a CPU 108 is connected to an RF demodulator 102 and digital demultiplexer 
106. The CPU 108 directs demodulation and demultiplexing of the proper 
charmel and data stream to obtain the correct video signal. Preferably, switches 
occur at an "I" frame if MPEG2 compression is used. The proper channel is 
determined either by examination of the user's input from user interface 130 
and /or any other information or criteria (such as personal profile information) 
stored in RAM/ROiM 120. For example, the RAM/ROM 12Q could store 
commands provided within the video signals as discussed in patent No. 
4,602,279, and incorporated herein by reference. The user interface 130 may be 
an infrared, wireless, or wired receiver that receives information from a user 
interface unit. 
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The RF demodulator 102 is part of the receiver, and demodulates data 
from the broadcast channel directed by the processor 108. After the data stream 
is demodulated, it passes through a forward error correction circuit 104 into a 
digital demultiplexer 106. The demultiplexer 106 is controlled by 
microprocessor 108 to provide a specific video signal out of a number of video 
signals which may be located within the data stream on the demodulated 
broadcast channel. The demultiplexed video signal is then decompressed and ^ 
decoded by decompressor/ decoder 110. The video signal is synchronized by a 
sync add circuit 150 and a sync generator 140. The video signal is then buffered 
by a video buffer 160. The buffered video signal is modulated by a modulator 
170 into a NTSC compatible signal. 

By using a video frame buffer 160 and delaying the viewing of a given 
signal, enough time is allowed for the decompressor/decoder 110 to lock onto, 
decompress, convert to analog, and wait for the resultant vertical interval of a 
second video signal. For example, assume video signal A is currently being ( 
processed and transferred through the circuit shown in Figure 6 and displayed. 
Based upon a user selection, the microprocessor 108 directs the digital 
demultiplexer 106 and RF demodulator 102 to switch to another video signal, 
video signal B. To accomplish this, the analog video from the first digital 
video signal, video signal A, complete with video sync, is fed into video frame 
buffer 160. This buffer 160 can hold the full video picture for "n" number of 
frames after which the signal is output to the display. In effect, a delayed video 
signal A is viewed "n" number of frames after the signal has been received. 
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When the user selects a different video path by means of pressing a button on a 
keypad or entry by other means, the microprocessor 108 instructs the digital 
demultiplexer 106 to stop decoding signal A and lock onto signal B to begin 
decoding signal B instead of signal A. 

While this is happening, even though the decompressor/ decoder 110 is 
no longer decompressing video signal A, the display is still showing video 
signal A because it is being read from the buffer 160. As soon as decompressing 
and decoding occurs, the microprocessor 108 looks for the next vertical 
blanking interval (VBI) and instructs the video frame buffer 160 to switch to its 
input, rather than its buffered output at the occurrence of the VBI. 

Since the RF demodulator 102, forward error corrector 104, digital 
demultiplexer 106, and decompressor /decoder 110 require a certain time period 
to decompress and decode the video signal B frame from its data stream, the 
size of the buffer 160 has to be large enough so that this processing can take 
place without interruption during the switching of the video signals. If 
desired, the system may continue to use the buffer 160 in anticipation of a 
future switch. By using the microprocessor 108 to manipulate the fill and 
empt}' rate of the buffer 160, the buffer 160 may be rapidly filled with video 
signal B frames and then after a period of time will be reset and ready to make 
another switch to another video in the same manner. The buffer 160 may also 
be reset by skipping frames or providing a delay between sequential frame 
outputs for a short time in order to fill the buffer. If a delay is used to maintain 
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video signal or frame output while the buffer 160 is being filled a slight 

distortion may occur for a brief amount of time. 

Because a first video signal is always displayed as the output of the buffer 

160 after the delay, the buffered video masks the acquisition and decoding of a 
second video signal. As long as the buffer 160 is large enough to keep the first 
video running while the second video is being decompressed and decoded, a 
seamless switch will occur. 

Figure 7 shows an alternate, dual tuner embodiment for seamless 
switching between separate video signals. In this embodiment, the 
microprocessor 108 controls the selection of the RF channel that is 
demodulated by RF demodulators 102A, 102B. The demodulated data streams 
enter the forward error correctors 104A, 104B. At the output of the forward 
error correctors 104A, 104B, the data streams are transmitted to the input of the 
digital demultiplexers 106A, 106B. 

As with the RF demodulators 102A, 102B, the digital demultiplexers (" 
106A, 106B are controlled by the microprocessor 108. This configuration allows 
the microprocessor 108 to independently select two different individual time- 
multiplexed video signals on different channels and data streams. If all the 
video signals of an interactive program were contained on a single channel or 
data stream, it would only be necessary to have a single RF demodulator, 
forward error corrector, and digital demultiplexer serially connected and 
feeding into the two digital video buffers 164, 165. 



aNSOOCID: <WO 9729458*1 J_> 



wo 97/29458 



PCTAJS97/02062 



21 

Two data streams are provided from the digital demultiplexers 106A, 
106B. One data stream carries video information pertaining to the video signal 
the user is currently viewing. The second data stream carries the video signal 
selected based on the user's previous and/or current interactive selections 
from the user interface, as determined by the microprocessor 108. 

The digital information on each of the two streams is buffered in digital 
video buffers 164, 165. The buffered signals are then decompressed and 
converted into analog signals by decompressors/ decoders llOA, llOB which 
include digital to analog converters. The decompressors llOA, llOB are 
preferably N4PEG2 decoders. 

A local sync generator 140 is connected to S}mc add 152, 153 and frame 
sync circuits 153, 154. Because both streams are synchronized based on signals 
from the same local sync generator 140, each stream becomes synchronized to 
the other. In particular, the signals on each stream are frame synchronized. 

A vertical blanking interval (VBI) switch 180 is connected to the 
microprocessor 108 so that the input may be switched during the vertical 
blanking interv^al of the current stream, resulting in a seamless switch to the 
viewer. 

The embodiment of Figure 7 operates as follows. Based on user 
responses and control codes, it is assumed that the microprocessor 108 
determines that a switch from video signal A to video signal C should be 
performed. RF demodulator 102A and digital demultiplexer 106A are 
processing the currently viewed video signal, video signal A, which is 
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progressing through the upper branch components. A command is issued 
from the microprocessor 108 to the RF demodulator 102B commanding a 
switch to the channel and data stream on which video signal C is located The 
microprocessor 108 also instructs the digital demultiplexer 106B to provide 
video signal C from the received data stream to digital video buffer 165. 

At this point, the upper RF demodulator 102A and digital demultiplexer 
106A are still independently receiving and processing video signal A, which 
continues through the upper branch of the circuit. 

At a certain point, the digital decompressor/decoder HOB in the lower 
branch will begin filling up with video C frames. After the video signal C is 
decompressed and decoded, it is converted into analog. A local sync generator 
140 inserts both local sync and frame sync to video signal C via sync add circuit 
152 and frame sync circuit 154 in order to synchronize it with the currently 
displayed video signal A, which is still being provided from the upper digital 
15 video buffer 164. At the appropriate switch point, triggered by programming 
codes supplied with each video signal A and C, the microprocessor 108 directs 
the VBI switch 180 to switch in the vertical blanking interval from video A to 
video C at which time video C will then seamlessly appear on the computer 
screen. 

20 Digital video buffers 164, 165 may be used in the circuit of Figure 7, but 

are optional. However, in an alternative embodiment the buffers, 164, 165 
would be required to provide a seamless switch if the Figure 7 circuit was 
modified to incorporate a single RF demodulator 102, single forward error 
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corrector 104, ar\d single digital demultiplexer 106 (as in Figure 3), each with a 
single input and single output. In this alternative embodiment, the circuit 
cannot independently receive and demultiplex two data streams on different 
frequency channels. One buffer is used to store previously received video 
5 signals, while the other buffer quickly passes through the selected video 
signals. ^ 
r Based on the same assumptions above, video signal A is progressing 

through the upper branch of the circuit and it is desired to switch to video 
signal C. However, in this alternative embodiment, the digital video buffer 
10 164 is providing maximum buffering to video signal A. 

Because it is desired to switch to video signal C, the microprocessor 108 
directs the alternative circuit (containing a single RF receiver 102, single 
forward error corrector 104 and single digital demultiplexer 106 connected in 
serial), to receive and demultiplex the data stream on which video signal C is 
15 located, which may be different than that of video signal A. When video 
signal C is demultiplexed, the microprocessor 108 directs the digital video 
buffer 165 to provide minimum buffering of video signal C so that 
decompressor /decoder HOB may quickly decompress and decode the digital 
signals. After decompression and decoding, video signal C is synchronized 
20 with video signal A. At this time video signal A is read for display from digital 
video buffer 164. The upper digital video buffer 164 must be large enough to 
provide video frames for output during the time it takes the RP demodulator 
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and digital demultiplexer to switch to video signal C and the time required for 
decompression, decoding, and synchronization of video signal C. 

When video signal C is synchronized with video signal A, the 
microprocessor 108 directs VBI switch 180 to switch from video signal A to 
video signal C in the vertical blanking interval of video signal A, thereby 
providing a seamless and flicker-free switch. 

At this time, digital video buffer 165 will begin to utilize maximum 
buffering by altering its fill/empty rate as described above with respect to the 
Figure 7 embodiment. When adequate buffering is achieved, a switch to 
another video signal may be performed in the same manner as described 
above. 

Another preferred embodiment is shown in Figure 8. This embodiment 
also includes an RF demodulator 102, a forward error corrector 104, and a 
digital demultiplexer 106. However, the circuitry differs along the rest of the 
chain to the television set or monitor. In this embodiment, a memory 190 is 
incorporated and connected to the output of the demultiplexer 106 for storing 
the compressed composite digital video signal. The decompressor/decoder 110 
is inserted at the output of the compressed memory 190. The 
decompressor/ decoder 110 decompresses the digital signal, converts the signal 
to analog and forwards the analog signal to the RF encode, 170 for transmission 
to the monitor. Once the composite compressed digital video signal is fed into 
the compressed memory 190, the microprocessor 108 directs a pointer to be 
placed somewhere along the compressed digital video signal. Based on the 
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placement of the pointer, different frames and different segments of the 
composite digital video signal will be read from memory 190 for 
decompression and decoding. 

The different video signals are distinguished from one another because 
5 they are labeled, preferably by headers. Assuming that video signal A has been 
selected for play on the monitor, the compressed digital memory 190 fills up 
r with A frames. Assuming a switch to video signal C is desired, the 

microprocessor 108 directs the RF demodulator 102 and digital demultiplexer 
106 to begin filling the compressed memory 190 with video C frames. The 
10 decoder pointer begins to move down. As soon as a sufficient number of C 
frames have entered the compressed memory, the pointer will then jump to 
the beginning of the C frames. The C frames are then output into the 
decompressor/decoder where the digital frames are converted into an analog 
signal. 

15 The digital video is multiplexed in a series of easily identifiable packets. 

These packets may contain full compressed frames of video (I frames) or may 
include only the differences between full frames (B frames or P frames). 

To be able to reconstruct the full video images, the 
decompressor/decoder 110 needs to have a minimum number of I, P and B 

20 frames. The decoder 110 needs only one I frame to decode an image. 

Conversely, two prior Anchor frames ('Ts'' and "P's") are necessar)' to decode B 
frames. In order to decode P frames, the decoder 110 only needs one Prior 
Anchor frame. When the microprocessor 108 instructs the digital demux 106 
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to start sending packets from a different data stream there is no way to be 
certain that the next packet will be an I packet needed for decoding the second 
video stream. To avoid a breakup of the video images, which would occur if 
the decompressor/decoder 110 suddenly started receiving packets unrelated to 
5 the stream it was decoding, the microprocessor 108 starts to fill up the memor 
190 with video signal C packets until it is determined that a full sequence of I, B 
and P frames are available. The decoder, 110 should receive the last bit of the ^ 
last B frame in a given, GOP (Group of .Pictures) before the switch, in order to 
prevent glitches when decoding. Furthermore, the last B frame of the GOP 

10 must only be backward predicted, not forward predicted or bidirectional 

predicted. As soon as the valid sequence is in memory 190 the microprocessor 
108 moves the memory read pointer to the start of a valid sequence of C video 
signal packets so that the decompressor/decoder 110 can successfully decode the 
C signals. This results in a seamless switch from video signal A to video signal 

15 C. (' 

This embodiment requires a data channel for enabling a synchronous 
switch between a first video stream and a second video stream. This data 
channel comprises the ACTV codes which link together the different program 
elements and information segments on the different video signals. In 
20 addition, the data channel also comprises synchronization pulses and a time 
code to signify to the pointer the proper time to skip from a memory location 
representing one video signal to a memory location representing another 
video signal in order to enable a seamless switch. 



aNSDOCID: <WO 9729459A l .!_> 



wo 97/29458 



PCT/US97/02062 



27 

The microprocessor 108 reads the data signal from the digital 
demultiplexer 106 and communicates pertinent data to the sync add circuit 
150, which is connected to sync generator 140. The microprocessor 108 is then 
able to synchronously communicate with the memory 190. 

• The time code sent will identify the timing for one picture, as well as for 
multiple pictures, and will lock the different pictures together. This is done 
through the use of similar clocks at both the transmission end and the 
receiver. A time code is used m order to keep the two clocks at both the 
transmission and receive end synchronously connected to one another. Once 
the clocks at both ends are working synchronously, each of the multiplexed 
video streams must be synchronized to the clocks. In order to synchronize the 
multiplexed video stream to the clocks, each of the individual channels must 
be referenced to a common reference point and must be identified. 

In the preferred embodiment, a packet header would be incorporated 
into the transport layer of the MPEG signal to identify the various channels. 
The packet header will also include information as to where to insert the 
verhcal blanking interval. In MPEG, the vertical blanking interval is not 
transmitted from the headend. Therefore, the vertical blanking interval must 
be generated locally. The packet header eye will identify at what time the 
vertical blanking interval is in existence in order to effectuate a seamless 
switch between analog pictures. 

In summary, the combination of clock and the information imbedded in 
either the transport layer of MPEG or in a separate packet on a separate data 
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channel effechiates the linking between each video signal and a corresponding 
time point. The data channel also includes information designating when all 
the various video signals will be in synchronism with one another. It is at 
these points that the microprocessor 108 may direct the pointer to skip from 
one location to another location, at a time (such as during the VBI) when a 
seamless switch will result. 

r 

Trigger Point,'? 

Interactivity is further enhanced in the interactive computer 
workstation embodiments through the application of trigger points 900 
scattered at various predetermined times throughout the program, a timeline 
representation of which is shown in Figure 9. The trigger points 900 
correspond to times when interactive events are scheduled to take place. 
These interactive events could be the selection and playmg of video, audio 
segments or the display of graphics. While the choice of particular video, (_ 
audio or graphics is still dependent on viewer selections, the viewer selections 
in response to displayed graphical interrogatory messages are preferably made 
during a period at the onset of the program or when a viewer first tunes into 
the program. These viewer selections are then utilized as inputs to macros 
called up at later times during the program by the controller upon the 
occurrence of the trigger points, identified to the interactive computer by 
unique codes embedded in the video signal. 
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The trigger points correspond to the times when the conventional 
program content can be altered and personalized for those subscribers capable 
of receiving the interactive signal. The programmer can place the trigger 
points at any time throughout the program. Since the trigger points are 
5 unknown to the subscriber, the subscriber does not know when they will 

receive a personalized message. In other words, an interactive response can 
( either immediately follow a corresponding user selection made to an 

interrogatory message or occur at a later time corresponding to a trigger point, 
or any combination of the two. Of course, timing of the interactive events 
10 should correspond to suitable times in the program where branching to 

interactive elements is sensible and does not clash with the program content of 
the conventional video still displayed on the television or other display 
monitor. 

At the onset of a trigger point 900, the controller will select one of 
^' 15 several possible audio (or video or graphic display) responses for presentation 

to the subscriber. As mentioned above and shown in figure 9, some of the 
responses may comprise a branch to either a video segment and /or audio 
segments. 

In combination with the use of trigger points 900, the present invention 
20 allows for the viewer to select certain options at the onset of the program to 
suit the viewers' preferences. For example, if the program broadcast is a live 
sports event, at an early trigger pomt 900, the viewer could be queried as to 
whether the viewer would prefer to receive audio in English, Spanish, French, 
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or perhaps hear the local announcer instead of the network announcer. Upon 
the viewer selection, the CPU directs a branch to the appropriate interactive 
segment. 

Each trigger point is identified preferably through the broadcast of ACTV 
codes sent as pari of the composite interactive program signal. The codes 
preferably include, at a minimum, the following information: (1) header 
identifying the occurrence of a trigger pomt; (2) function ID (e.g., selection of ^' 
audio or graphics responses, etc.); and (3) corresponding interrogatory 
message(s). The first bit sequence simply identifies to the controller that a 
trigger pomt is about to occur. The function ID designates the macro or other 
set of executable instructions for the controller to read and interpret to obtain 
the desired result, e.g., a selected video and/or audio response. 

Upon extraction of the codes by the data decoder, the CPU 108 reads and 
interprets the codes and calls from memory a particular user selection(s) 
designated by the trigger point codes. The user selections correspond to C 
subscriber answers to a series of interrogatory messages preferably presented at 
the beginning of the program. After obtaining the appropriate user 
selection(s), the controller 108 reads and performs the executable instructions 
usmg the user selection(s) as input(s) in the macro algorithm. The result of the 
algorithm is either a selected video stream, audio and/or selected graphics 
response. The video/audio response can be called from memory if it is 
prestored, called from external data storage, or the controller can command the 
switch to branch to the particular video audio stream if the response is 
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broadcast concurrently with the trigger point. After the selected video/audio 
response is played to the subscriber, the switch branches back to the standard 
program, shown at time t^ in figure 9. 

As mentioned above, a series of interrogatory messages are preferably 
presented when the subscriber begins watching the interactive program. These 
interrogatory messages can be presented in any one of three ways. First, the 
interrogatory messages can be presented as graphics displays overlaid by the 
interactive computer workstation onto a video signal, wherein the graphics 
data is sent in the vertical blanking interval of the composite interactive signal, 
or alternatively stored on the hard disk or external storage. Second, the 
interrogatory messages are presented as graphics displays as discussed above, 
except the graphics data comes from local storage, external data storage (e.g., CD 
ROM, cartridge, etc.), or a combination of data in the VBI and data called from 
either local or external data storage. Third, graphics data can be presented in 
the form of user templates stored at the interactive computer workstation. 

User selections corresponding to answers to the n successive 
interrogatory messages are received by the remote interface at the beginning of 
the show, stored in memory and used throughout the show at the appropriate 
trigger points to subtlely change program content as the show progresses. 
Preferably, each interrogatory has a set of possible answers. Next to each 
possible answer will be some identifier corresponding to a label on a key on the 
user interface.- The subscriber depresses the key corresponding to their answer 
selection. This selection is decoded by the remote interface and controller, 
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stored in raernory. preferably RAM, and used la.er as required by an algorithn, 

designated at a trigger point. 
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While such interacHve programing n,ay include a plurai.iy of v.deo 
signals, the in.eracHve mult,n,edia computer work station 6, descnbed het.in 
may also provide for personalized audio interactivity by way of a single 
standard video and audio televis.on s.gnal w.th a plurai.ty of add.tiona, audio 
Signals and/or graphics data for providing interactivity, as shown in Figures 
10-13. The interaction with the subscribers comes pr.marily by way of selection 
of one or more linked audio segments from a plurality of audio segments 
Whereby the selected audio segments) are chosen as a function of previous 
user responses. Interactivity ,s enhanced thK,ugh the use of overlaid graphics 
displays on the video which like the audio responses, also vary according to 
selections made by the subscriber on the user interface. Audio segments are 
used to provide personalized responses to subscriber selections. The graphics, 
on the other hand, are used to both query the subscriber, preferably at the 
beginning of the program, and also to provide personalized graphical messages 
to subscribers. The .n.eraCve show also composes control data for controlling 
the interactive computer work station. 

Multiple audio segments forming the set of suitable responses to 
interrogatory message can be sent as part of a standard video signal. There 
a number of different ways to effectively forward the necessary audio segments 
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for a given interactive event to the interactive computer. The interactive 
elements may be broadcast synchronously (alternative responses aligned in 
time), serially, on separate channels, embedded in the existing video and/or 
transmitted before or during the program. Audio segments tagged for a given 
5 interactive event, can be sent to the interactive computer work stations much 
earlier than the scheduled event during the program, in which case the 
segments are preferably stored in temporary memory, or the segments can be 
transmitted concurrently with the event.. With the present invention, it 
makes no difference how the audio segments reach the interactive computer 

10 work station as long as they are available for selection at the computer 6 at the 
predetermined "trigger points," described below. For example, the audio 
segments could also be stored in local external data storage such as CD-ROM. 

In one preferred "trigger point" embodiment, interactive audio shows 
can be delivered in the standard television channel. In this embodiment, four 

15 audio responses are available at each trigger point, however, only two audio 
charmels need be broadcast, or otherwise input, to the interactive computer 6. 

This embodiment has the advantage of requiring merely one television 
channel. Channel 1 is the "home" channel. When channel 1 is playing, 
channel 2 is used to download the audio for tracks 3 and 4 to the interactive 

20 computer 6. This downloaded audio is stored as wave files in the local unit. 
When it is time to branch, audio tracks 1 and 2 are played on the two audio 
input channels, while tracks 3 and 4 are generated from the audio wave files 
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on the interactive computer 6. A seamless branch is made from any one of 
these channels to any of the other channels. 

Figure 10 shows an overview of a preferred interactive computer work 
station embodiment. Other digital and audio alternative embodiments for the 
provision of audio interactivity are shown in figures 6-8 of U.S. Patent 
Application Serial No. 08/289,499, herein incorporated by reference. The 
embodiments represent different apparatus for receiving, processing and ^ 
stonng the alternative mteractive audio segments which are received in 
different transmission formats. With these embodiments, the interactive 
systems are no longer solely limited to selecting audio from multiple parallel 
tracks of audio, related in time and content, nor is the interactive questions- 
immediate answer format, as disclosed m previous patents, necessary. Of 
course, the systems of the present invention can still use the question- ' 
immediate answer format or a combmation of such format and delayed 
response via trigger points. Hie concept remains the same, i.e., to select audio C 
responses which are matched to user selections by some function. 

The elements of the audio interactive embodiment can be incorporated 
and provided by the interactive multi-media work station. Preferably, this 
configuration comprises a video demodulator board, a keypad for entering 
subscriber selections, an extractor board for separatmg the audio signals and 
data from the conventional video signal, temporary and permanent data 
storage, a modem (optional), audio switch and a processor. 
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Referring to a preferred embodiment shown in figure 10, the video 
demodulator 616 outputs the standard video signal which is transported to a 
Gen-lock circuit 623 and character generator 624 as wells as to a voice /data 
extractor 174. At the output of the Gen-Lock circuit 623 and character generator 
5 624, the video is forwarded via the RF modulator 622 to the television or 
computer display monitor. The processor 178 preferably controls an n X 1 

( : switch 620, the output of which is an appropriate audio segment to be sent to 

the television set for presentation to the subscriber. Of course, the switch could 
have more than one output, in which case more than one viewer can watch 
10 the video on the same monitor and each receives individualized audio 

response through the use of headphones. The processor 178 sends a command 
to the audio switch 620 to disconnect the standard audio at the beginning of an 
interactive segment. The extraCtor,174 essentially reverses the process by 
which the audio and data signals were inserted into the video signal. As 

^- 15 explained below, the voice/data extractor 174 removes the additional audio 

segments and data that are hidden in the standard video signal. The data is 
forwarded to the microprocessor 178 and the audio segments get sent either to 
an audio switch 620 or to temporary memory 202 depending on where the 
instructions teach the segments to be forwarded, all of which occurs under the 
20 control of the microprocessor 178. The microprocessor 178 reads and interprets 
the instruchons either broadcast in the data codes or resident in the operating 
software at the interactive work station 6. 
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The microprocessor 178 interprets the extracted data as either control 
data, including instructions for switching between voice channels, or graphics 
data for on screen display. If the data is on-screen display data, the data is 
preferably prefixed by a command designating the data as on-screen display 
data, as opposed to control data. In the preferred embodiment, the controller 
178 also examines the control data for the occurrence of a header code 
designating the onset of a trigger point in the program. 

If the trigger point codes designate a macro which calls for the placement 
of a graphics display on the video, the microprocessor 178 reads the codes, 
accepts any graphics data sent from the head-end, calls for and examines the 
actual bit maps stored in memory 284 or external memory 629 and designating 
the identity of the characters, and then commands the character generator 624^ 
to overlay particular characters at particular points on the screen. Therefore, 
the graphics are preferably generated locally with the bit maps stored in 
memory 289. The graphics are selected for presentation either in ( 
predetermined sequence, through the use of control codes in the composite 
interactive program, developed when the program was created at the 
operations center, or more flexibly through the execution of algorithms by the 
processor 178 utilizing stored subscriber selections to previous graphic 
interrogatory messages. The algorithms are preferably part of the operating 
systems software stored in memory at the interactive work station. 
Alternatively, the algorithms could be included in the data portion of the 
composite interactive signal. 
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The graphics can be utilized to overlay any portion of the screen of the 
television screen. The character generator 624 is locked by a Gen-lock circuit 
623 which allows for the synchronous placement of the graphics on the video. 
The character generator 624 is preferably a standard on-screen display chip 
5 which takes incoming video, locks the video and superimposes on the video 

the characters as instructed by the microprocessorl78. Specifically, the character 
generator 624is a switching system which takes the active lines of video and 
switches to a mode of sending the graphics characters for a predetermined 
time, and then switches back to the video when the character is finished being 

10 written on the screen. 

Because the graphics are generated locally, subscribers without the 
interactive multimedia computer 6 are not be able to view the graphics. For 
those subscribers possessing the interactive capability, the graphics can be used 
for both posing interrogatory questions to subscribers at the onset of the 

15 program, consistent with the trigger point embodiment, posing questions 

during the program, or used to provide a personalized response to previous 
individual subscriber selections. 

Preferably at the beginning of the program or when a viewer first times 
in, a series of interrogatory messages are presented to the subscriber. The 

20 subscriber responds to the interrogatory message by depressing a button via the 
user interface device corresponding to an answer selection listed on the 
interrogatory graphics screen. If the subscriber has made a selection using a 
remote, a signal is received by the IR interface 628 which processes the signal 
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and forwards the signal to the processor 178. TT.e processor preferably creates a 
packet comprising the user selection and a header code that identifies the 
particular interrogatory message associated with user selection and sends the 
packet to memory 284. Each user selection to each interrogatory is stored in 
this fashion. These selections will be called later in the program at appropriate 
times when identified by the trigger point codes and then used in macros or 
algorithms to determine interactive audio and /or graphics responses. < 

The presentation of the graphics interrogatory messages can also be 
made a function of subscriber selections to previous interrogatory messages. 
The logic used in the codes for selecting the next graphics message is similar to 
that used for selecting audio messages. One method, as disclosed in earlier 
ACTV patents, is the ^decision tree" logic methodology. The subscriber makes 
a selection to a first predetermined interrogatory graphics message.' After the 
subscriber hears an appropriately branched audio channel, the processor 178 
will interpret graphics algorithmic codes sent down from the operations center ( 
608 and will read from memory 284 an appropriate next graphics message. The 
processor 178 then directs the character generator 624to overlay the selected 
graphics message onto the next frames of video. 

The advantages discussed above in relation to presenting an interactive 
program using trigger points are obtainable in each of the interactive computer 
embodiments shown in figures 11-13. In the embodiment shown in figure 11, 
alternative audio.segments are preferably sent serially from the operations 
center in the SAP channel. The demodulator 617 receives a composite 
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interactive signal comprising the standard video and standard audio signal 
along with an audio subcarrier. The demodulator 617 breaks the signal into it's 
component parts, forwarding the baseband video to a data extractor 175 and the 
standard audio to an audio switch 620, The line 21 data extractor 175 takes out 
5 the data codes, including the trigger points. 

The SAP channel comprises a plurality of audio segments lined up 
serially. The audio segments are digitized in the analog to digital converter 
750 and are preferably stored in digital audio memory 283. At certain times 
during the program, data codes will designate a trigger point and key the 

10 microprocessor 178 to select and play an audio segment corresponding to 
previous. user input(s), according to the process described above. The 
microprocessor 178 calls the appropriate audio segment(s) from internal 
memory or external data storage 629 and commands the audio switch to pass 
the selected audio segment to the RF modulator 622 for play to the subscriber. 

15 At the end of the interactive time period, the controller 178 instructs the audio 
switch 620 to again pick up the standard audio. 

In an alternative embodiment similar to that as shown in Figure 11 and 
discussed above, the simple addition of a second tuner, receiving the 
composite RF signal, could be used to tune to a second audio channel for 

20 collection of transmitted audio segments. The tuner would pass the audio 

segments to the A/D converter with the operation of the rest of the interactive 
computer workstation similar to that described above in connection with 
Figure 11. 
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Figure 12 shows another interactive computer workstation embodiment 
for providing alternative audio and graphics segments. This embodiment uses 
two tuners: an RF demodulator 616 and a data tuner 615. The RF demodulator 
616 tunes to and demodulates the conventional video and audio signal in the 
standard video bandwidth. The data tuner 615 receives a single digital audio 
signal. The signal comprises digital serial audio segments modulated onto an 
analog carrier. The data tuner 615 demodulates the signal into digital audio. C 
The digital interface selector and error corrector 177 separates the audio 
segments and performs error correction according to any error correction 
scheme commonly understood in the art. The controller 178 directs the 
selector 177 to extract selected digital audio segments from the serial digital 
stream and send them to the digital audio memory 283. Selection of one or 
more audio segments for play as personalized messages on the speakers occurs 
according to the processes described above. After the controller 178 commands 
the memory 283 to forward a digital audio segment, the segment is converted ( 
to analog by the digital to audio converter 176 and is subsequently passed to the 
RF modulator 622 for play on the speakers. 

Another interactive computer 6 workstation embodiment for receiving, 
storing and selecting alternative audio segments is shown in figure 13. At the 
operations center, the audio segments are digitized, time division multiplexed, 
modulated and converted to frequencies in unused channel frequency space in 
the cable television spectrum, e.g., cable guard bands. 
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The RF demodulator 616 again demodulates the conventional video 
and audio signal. The data extractor 175 receives the signal and extracts the 
VBI line 21 data codes. The data in the VBI indicates the frequency charmels in 
which the digital audio segments are transmitted. For example, audio 
5 messages A-E are located in between channels 14 and 15. The controller 178 
instructs the data tuner 615 to tune to that part of the spectrum between 
channels 14 and 15. Alternatively, an autotune capability can be used to find 
the audio channels in the spectrum. 

The tuner 615 demodulates the digital audio signal and forwards the 
10 signal to the digital demultiplexer 700. The demultiplexer 700 demultiplexes 
the signal into n digital audio channels and forwards each channel to a 
separate D/A converter 702-710 where each of the digital channels are 
converted to analog audio. As described above, one of these channels can be 
selected as identified at the trigger points for play over the audio speaker to the 
15 subscriber. 

The embodiments described above and shown in connection with 
figures 10-13 relate to different ways of receiving broadcast audio segments. 
Alternatively, interactive audio segments, or graphics elements, could be 
prestored on cartridge, CD ROM, an audio card, or even floppy disk. 
20 Even more enhanced and flexible operation can occur through the 

addition of external data storage, such as CD ROM or cartridge. For example, 
sports statistics or other information on athletes or others can be stored in CD 
ROM. During live sports event either audio segments or graphics displays 
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focusing on the athlete can be called by the processor and presented to the 
viewer as a function of user selection of an option or at a trigger point if the 
user indicated during queries at the beginning of the live event that they were 
interested in a particular player. 

5 

Memory 

The interactive computer also has the advantage of remembering 
subscriber responses and using these responses in choosing a video/audio 
response, and /or graphics interrogatory message, to present to the student. 
10 Memory branching is a technique of the present invention where the 
algorithm assembles video/audio responses and graphics interrogatory 
. messages according to the current and previous user inputs. Memory 
branching is accomplished by linking video/audio streams and/or successive 
graphics mterrogatory messages together in a logical relationship, as described 
15 in U.S. apphcation no. 08/228,355, herein incorporated by references. In this 
scheme, the interactive computer processor contains logic (preferably, in the 
software algorithm) and memory to store previous subscriber selections and to 
process these previous responses in the algorithm to control future 
video/audio stream selection, as well as future graphics message selection. 



20 



User Prnfily «; 

In a preferred embodiment, the interactive computer can have stored 
its memory a "user profile." The "user profile" preferably contains 
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characteristics of the particular viewer at that subscriber location, such as sex, 
hobbies, interests, etc. This user profile is created by having the user respond to 
a series of questions. Alternatively, the user profiles could be created at a host 
and sent to the interactive computer over a network. This information is then 

5 used by the interactive computer software to create a compendium of the 

viewer's interests and preferences - i.e., a user profile. The stored user profile 
would be used in place of the question /answer format, and thus, dictate the 
branches to interactive segments of interest to the viewer. 

Alternatively, the interactive computer 6 can be programmed to create a 

10 user profile of each viewer based on the selections made during one of the 

interactive programs. Furthermore, such a user profile could be modified or 
enriched over time based on selections made during future interactive 
programs. For example, the' 'memory' technique described above can be used 
to modify the user profile based on user response over time. 

15 Once the profile is created, the programming choices or interactive 

responses can be triggered based on the content of the user profile itself. For 
example, if the user profile suggests that the viewer is particularly interested in 
sports cars, a sports car commercial could be played for the viewer at a 
predetermined point in the program. As another application, if a viewer's user 

20 profile indicates that the viewer is interested in cooking, whenever the viewer 
watches such a program, the user profile would trigger the interactive program 
to download recipes and either display such recipes on the screen or send the 
recipes to an attached printer. 
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Applications 

The embodiments, described above, allow for several possible 
applications. For example, in a live sports event, one channel could carr>' the 
standard video channel, with other channels carrying different camera angles 
and /or close-ups of particular players. 

Audio interactive applications include the recording of audio clips for 
each player in the game. In this application, the viewer may access a pull- ^ 
down menu, where he can choose a name of a particular player in the game. 
When this selection is made, the appropriate audio segment is called from 
memory and played for the viewer. In a similar manner, statistics in the form 
of text and graphics can be displayed for a selected player. 



Internet Applications 

Interactive programs of the present invention can be created using the 
15 Internet. Interactive program authors can access a particular Internet site and C 
download graphics, audio and video clips and suggested interactions. The 
author can then use these elements in the authoring tools to create an 
interactive program. 

Furthermore, viewers can watch interactive programs from the Internet 
itself using the systems of the present invention. From an Internet site, 
viewers can access a single channel interactive program, such as described 
above. The viewer would watch the video on his or her computer, while the 
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audio and/or text/graphics from Web site locations, for example,would be 
presented as a function of his or her specific choices via interactive commands. 

In addition, viewers can choose between multiple video streams 
originating from a site on the Internet. The seamless branching between 
5 different video streams would occur through interactive commands resident 
in the viewer's computer. 

Using the foregoing embodiments, methods and processes, the 
interactive multmedia computer maximizes personalized attention and 
interactivity to subscribers in their homes in real time. Although the present 
10 invention has been described in detail with respect to certain embodiments 

and examples, variations and modifications exist which are within the scope of 
the present invention as defined in the following claims. 
We claim: 

r 
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1. An interactive computer work station for presenting an integrated 
multimedia interactive presentation, comprising: 

a means for displaying an integrated audio, graphics and video 
presentation, the presentation comprising a common audio and common 
video signal, and at predetermined times, at least two selectable user options ^ 
corresponding to an interrogatory message; 

a means for interacting with the integrated presentation, wherein the 
user is prompted to interact with the integrated presentation in response to the 
interrogatory message by selecting one of the user selectable options; 

a means, connected to the interaction means, for determining an 
appropriate personalized feedback response, wherein the feedback response 
may consist of video, audio and graphics segments and the selected feedback 
15 response is based on one or more user selected options; and 

a means, connected to the determination means, for presenting the 
appropriate personalized feedback response to the viewer, wherein the 
appropriate personalized feedback response may occur immediately after user 
selection or at a later predetermined time in the integrated presentation. 



20 



2. The interactive computer work station of claim 1, wherein the 
interactive computer work station further comprises: 
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a plurality of video sources, each source storing a separate and distinct 
video segment, v^herein the common video signal contains data commands; 

a means, connected to at least one video source, for extracting data 
commands, wherein the data commands comprise branching codes; 
wherein the determination means comprises: 

a means for detecting the branching codes; and 
a means, cormected to the detection means, for processing the 
branching codes, wherein the personalized feedback response is based on 
the branching codes and the user selected options. 

3. The interactive computer work station of claim 1 further comprising: 
an audio card, connected to the determination means, for storing the ■ 

plurality of different audio segments. 

4. An interactive computer work station for presenting an integrated 
multimedia interactive presentation, comprising: 

a means for receiving a common video signal and a plurality of different 
video segments, wherein the common video signal has embedded data 
commands; 

a means for displaying an integrated audio, graphics and video 
presentation, the presentation comprising a common audio and the common 
video signal, and at predetermined times, at least two selectable user options 
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corresponding to an interrogatory message, the interrogatoiy message 
presented as part of the video signal; 

a means, connected to the receiving means, for extracting data 
commands, wherein the data commands comprise branching codes; 

a means for interacting with the integrated presentation, wherein the 
user is prompted to interact with the integrated presentation in response to the 
interrogatory message by selecting one of the user selectable options; 

a means, connected to the interaction and extracting means, for 
determining an appropriate personalized feedback response, wherein the 
feedback response may consist of video, audio and graphics segments and the 
selected feedback response is based on one or more user selected options and 
the branching codes; and 

a means, connected to the determination means, for presenting the 
appropriate personalized feedback response to the viewer, wherein the 
appropriate personalized feedback response may occur immediately after user 
selection or at a later predetermined time in the integrated presentation. 



5. The interactive computer work station of claim 4 further comprising: 
an audio card, connected to the determination means, for storing the 

20 plurality of different audio segments. 

6. An interactive computer work station for presenting an integrated 
multimedia interactive presentation, comprising: 



15 
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. a means for receiving a common video signal and a plurality of different 
video segments, wherein the common video signal has data commands; 

a means for displaying an integrated audio, graphics and video 
presentation, the presentation comprising a common audio and the common 
video signal, and at predetermined times, at least two selectable user options 
corresponding to an interrogatory message, the interrogatory message 
presented as part of the video signal; 

a means, connected . to the receiving means, for extracting data 
commands, wherein the data commands comprise branching codes and trigger 
points; 

a means for interacting with the integrated presentation, wherein the 
user is prompted to interact with the integrated presentation in response to the 
interrogatory message by selecting one of the user selectable options; 

a means, connected to the interaction and extracting means, for 
determining an appropriate personalized feedback response, wherein the 
feedback response may consist of video, audio and graphics segments and the 
selected feedback response is based on one or more user selected options and 
the branching codes; and 

a means, connected to the determination means, for presenting the 
appropriate personalized feedback response to the viewer, wherein the 
appropriate personahzed feedback response occurs upon the occurrence of one 
of the predetermined trigger points. 



9729458A1 I >" 



wo 97/29458 



PCT/US97/02062 




wo 97/29458 



PCT/US97/02062 



2/13 




BNSDOCIO: <V/0 972945BA1J,> 



wo 97/29458 



PCT^S97/02062 




wo 97/29458 



PCT/US97/02062 




wo 97/29458 



5/13 



PCTAJS97/02062 




&NSDOCia<WO 9729458A1 I > 



wo 97/29458 



PCT/US97/02062 



6/13 



o 



3 

o 



e 


0 ' 
» 


p 
cr 







r 








0 












a. 












H 3 



r 

o 

< 



BNSDOCID: <W0. 



.9729458A1J_> 



wo 97/29458 



PCT/US97/02062 




wo 97/29458 



PCT/US97/02062 



8/13 



r 



T 






! 






















4 


- 



c. 






BNSDOCID: <WO 



972945aA1 1 > 



wo 97/29458 PCT/US97/02062 

9/13 





BNSDOCia <V/0 97294S3A1 I > 



wo 97/29458 PCT/US97/02062 

10/13 




BNSOOCID; <W0 



97294S8A1 I > 



wo 97/29458 



PCT/US97/02062 



11/13 




BNSOOCID. <WO 972945aAl_L> 



wo 97/29458 



PCT/US97/02062 




BNSDOCIO: <WO 972945SA1 I > 



wo 97/29458 



PCT/US97/02062 




So. I 



3 o 
Q 2 



B.NSOOCID. <V/0 9729458A1 I > 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/i;S97/02062 



A. CLASSinCATlON OF SUBJECT MATTER 
IPC(6) :G06T 13/00 

US CL iPleaae Soe ExUa Sheet. 
According to IntemationaJ Patent CUatification (IPC) or to both national claaiification and IPC 

B. n ELDS SEARCHED 

Minimum documeniation searched (ciaatiiication system followed by cUsatfication symbols) 

U.S. : 395/806, 807, 762. 173. 174, 327, 328; 434/307R. 308; 364/514A; 348/6. 7. 8 



Documentation searched other than minimum documentation to the- extent that such documcnu are included in the fields searched 



Electronic data base consulted during the iniemationai search (name of data base and, where practicable, search terms used) 



c 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 



X 

Y 

A,P 
A 
A 
A 



Citation of document; with indication, where appropriate, of the relevant passages 



US 5,454,722 A (HOLLAND ET AL) 03 October 1995, coL2, 
lines 30-45, col. 3, lines 28-36, coL4, line 40, col. 5, lines 7- 
29; F1G.4 

US 5,541,662 A (ADAMS ET AL) 30 July 1996, col. 2, line 
59, coL4, lines 5-46, col-7, line 36, coL8, lines 51 to col. 9, 
line 24. 

US 5,557,724 A (SAMPAT ET AL) 17 September 1996. 
US 5,033,969 A (KAMIMURA) 23 July 1991. 
US 5,109,482 A (BOHRMAN) 28 April 1992. 
US 5,388,197 A (RAYNER) 07 February 1995. 



Relevant to claim No. 



1 

2-6 

2-6 

1-6 
1-6 
1-6 
1-6 



fx] Further documcnu atc listed in the continuation of Box C. [ | Sec patent family annex. 



Spccml calof orie* of eked Jnnmrali' 

< U*f%mitmt deftuDg geaenJiMe of ibewt wkidi ia i 
IP W of piuticitW rrWifnr 

naMoc ptMit^md om or WW the mttnmiioaml fUkif date 



hier ^^^r^^n*^ pubUkkked mfSer tbc ■aiCTWficwMj fUioc date or fwi o ik y 
Kvi Bo( io cooA«c< with ifac •f>pLicat*oo bul cited lo uoderateod tfke 
priac^pic or ^eory Modcriyn^ (be mvcntjoo 

doamacol of partioular rcicva/toc; tikc c Waned invcDboo cmiwxj* be 
ml DOvei or OMOaot be oocMidcred U> iavoKe %a iDvenlTvc Mcp 



doMMHC wkkb amy ikraw donkm om ffioriqr 
cM to t^mhikk Ike pOaintWw d^ of 



ckinif s) or wkkh ■ 



H wi— wi pu M a Ud prior to Ike kiermatkxMkj fUaa^ bMl Uler t 
Ike prioriiy d^le ck 



of puticukr relevMDoe: ibe cUkaed i 
I 10 kvoive IB BveBiivc mep wkaa ike 
wlik o*e or BMMe ocker auck docuBMoa, such cottbittaooo 
iout to e pence *kiUod m the afi 



Ann wm mi ■c m ke r of Ike amatc pmUmi frntaify 



Dale of the »ctua] completion of the international search 
15 APRIL 1997 



Name and mailing addreas of the ISA/US 
CommiMioottr of Pateou aad Tndecnarka 
Box PCT 

Waaiuo^vm, D.C. 20231 
Facsimile No. r703) 305-3230 



Date of mailing of the iniemationai search report 



02 mm? 



Form PCT/1SA/2I0 (second ahcctKJuly 1992)* 



uthori2jcd officer 

STEPHEN HONG ^V^ijCW 
Telephone No. (703) 30S-5465 



BNSDOCID: <W0 9729458A1_|_> 



INTERNATIONAL SEARCH REPORT 



Inicmationai applicAlion No. 
PCT/US97/02062 



C (Continuauon). DOCUMENTS CONSIDERED TO BE RELEVANT 



Citcgory* 



Citition of docuincnt, with indicAlion. where appropriate, of the relevant padsagea 



Rclcvam to claim No. 



A 
A 
A 



US 5,467,288 A (FASCIANO ET AL) 14 November 1995. 
US 5,261,820 A (SLYE ET AL) 16 November 1993. 
US 5,455,910 A (JOHNSON ET AL) 03 October 1995. 



1-6 
1-6 
1-6 



Fonii PCT/ISA/210 (continuation of teoopd sbeetKiuly 1992)* 

BNSDOCID <WO 9729453* I I > 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCTAJS97/02062 



C 



A. CLASSIRCATION OF SUBJECT MATTER: 

US CL : 

395/806. 807. 762. 173. 174. 327. 328; 434/307R. 308; 364/514A; 348/6. 7. 8 



c 



Form PCT/lSA/210 (cxtr* ihcetKJuly 1992)* 



BNSDOCID:<WO S7294SaAlJ : 



THIS PAGE BLANK (yspio) 



C 

i 
1 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of tlie Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the origin 
documents submitted by the appHcant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SffiES 
FADED TEXT OR DRAWING 



^BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning tliese documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 




